THE MAPPING CLASS GROUP ACTS REDUCIBLY ON SU(n)-CHARACTER VARIETIES

When G is a connected compact Lie group, and π is a closed surface group, then Hom(π,G)/G contains an open dense Out(π)-invariant subset which is a smooth symplectic manifold. This symplectic structure is Out(π)-invariant and therefore defines an invariant measure μ, which has finite volume. The corresponding unitary representation of Out(π) on L(Hom(π,G)/G, μ) contains no finite-dimensional subrepresentations besides the constants. This note gives a short proof that when G = SU(n), the representation L(Hom(π,G)/G, μ) contains many other invariant subspaces. Let G = SU(n) and π be the fundamental group of a closed oriented surface Σ. Let Hom(π,G) be the space of representations π −→ G. The group Aut(π) × Aut(G) acts on Hom(π,G). Let Hom(π,G)/G be the quotient of Hom(π,G) by {1}× Inn(G). Then Out(π) := Aut(π)/Inn(π) acts on Hom(π,G)/G. The Out(π)-action preserves a symplectic structure on Hom(π,G)/G which determines a finite invariant smooth measure μ (on an invariant dense open subset which is a smooth manifold). When G is compact, the total measure is finite (Jeffrey-Weitsman [6, 7], Huebschmann [5]). There results a unitary representation of Out(π) on the Hilbert space H := L(Hom(π,G)/G, μ). Let C ⊂ H denote the subspace corresponding to the constant functions. The following theorem is proved in Goldman [3] for n = 2 and Pickrell-Xia [8, 9] in general: Theorem. The only finite dimensional Out(π)-invariant subspace of H is C. According to [3], the only finite-dimensional invariant subspace of H consists of constants. Let H0 denote the orthocomplement of the constants in H, that is, the set of f ∈ H such that ∫ fdμ = 0. The goal of this note is an elementary proof of the following: Date: May 12, 2008. Partially supported by NSF grants DMS0405605 and DMS-0103889. 1 2 WILLIAM M. GOLDMAN Theorem. The representation of Out(π) on H0 is reducible. In general for a compact connected Lie group G, the components of Hom(π,G) and Hom(π,G)/G are indexed by π1(G ) where G is the commutator subgroup. In that case the ergodicity/mixing results of [3, 8, 9] imply that the only invariant finite dimensional subspaces are subspaces of the space of locally constant functions, a vector space of dimension |vertπ1(G )|. The method of proving reducibility requires a nontrivial element of the center of G. For simplicity, we only discuss the case G = SU(n) in this paper. I am grateful to Steve Zelditch, Charlie Frohman, and Jeff Hakim for helpful conversations. 1. The center of SU(n) Let Z ∼= Z/n denote the center of G, the group consisting of all scalar matrices ζI where ζ = 1. Then Hom(π,Z) ∼= H(M ;Z/n) acts on Hom(π,G)/G by pointwise multiplication: If ρ ∈ Hom(π,G) and u ∈ Hom(π,Z), then define the action u · ρ of u on ρ by: (1) γ u·ρ 7−→ ρ(γ)u(γ). Recall the definition [2] of the symplectic structure on Hom(π,G)/G. Suppose that ρ ∈ Hom(π,G) is an irreducible representation. By Weil [11] (compare also Raghunathan [10]), the Zariski tangent space to Hom(π,G) at ρ identifies with the space Z(π, gAdρ) of cocycles where gAdρ is the π-module defined by the composition π ρ −→ G Ad −→ Aut(g) and g is the Lie algebra of G. Since ρ is irreducible, G acts locally freely and Hom(π,G)/G has a smooth structure in a neighborhood of [ρ]. The tangent space to the orbitG·ρ equals the space of coboundaries B(π, gAdρ). The tangent space to Hom(π,G)/G at [ρ] identifies with the cohomology group H(π, gAdρ). Let u ∈ Hom(π,Z). Since Ad(Z) is trivial, the action of u induces an identification of π-modules gAdρ −→ gAd(u·ρ) and hence of tangent spaces T[ρ]Hom(π,G)/G = H (π, gAdρ) −→ T[u·ρ]Hom(π,G)/G = H (π, gAd(u·ρ)). The symplectic form ωρ at [ρ] is defined by the cup product H(π, gAdρ)×H (π, gAdρ) −→ H (π;R) REDUCIBILITY OF MAPPING CLASS GROUP ACTIONS 3 using the pairing of π-modules induced by an Ad-invariant nondegenerate symmetric bilinear form on g as a coeefficient pairing. Evidently the symplectic form ω, and therefore the corresponding measure μ, are Hom(π,Z)-invariant. 2. Trace functions Suppose that γ ∈ π. The function Hom(π,G) −→ C ρ 7−→ tr(ρ(γ)) is Inn(G)-invariant and defines a function Hom(π,G)/G tγ −→ C. We extend this definition to products of more than one element of π: Let γ = (γ1, . . . , γs) ∈ π . Define the trace function of γ as the product: tγ([ρ]) := tγ1([ρ]) . . . tγs([ρ]) Using the definition (1) of the action of Hom(π,Z) on Hom(π,G)/G, the action of u ∈ Hom(π,Z) on the trace function tγ is given by u · tγ, defined by: (2) ρ u·tγ 7−→ u(γ)tγ (